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Abstract Body 


Background / Context: 

Educators are increasingly concerned about the low level of mathematics performance of 
U.S. students on the TIMSS and other international assessments of mathematics (National 
Mathematics Advisory Panel, 2008) as well as their insufficient preparation for mathematics 
standards, such as the Common Core State Standards. Students from low-income and minority 
backgrounds demonstrate lower levels of mathematics achievement than their peers from more 
advantaged backgrounds, and there is compelling evidence that this SES-related achievement 
gap in mathematics emerges prior to school entry. Cross-cultural research on children’s early 
mathematical development in China and the United States has found that cross-SES differences 
within each country are present at 3 years of age. The SES gap, however, narrows in China 
during the preschool years but widens in the United States (Starkey & Klein, 2008). One 
contributing factor is that preschools in urban areas of China implement a mathematics 
curriculum for all children beginning at age 3, whereas many public preschool programs in the 
United States do not implement effective mathematics curricula (e.g., Preschool Curriculum 
Evaluation Research Consortium, 2008; U.S.DHHS/ACF, 2010). It is also known from a recent 
meta-analysis of several large longitudinal studies that children’s mathematics knowledge in 
kindergarten is the strongest predictor of their later school achievement - stronger than early 
literacy knowledge, attention skills, or socioemotional development (Duncan, et ah, 2007). If this 
gap is not addressed early, it persists and increases over time (e.g., Bodovski & Farkas, 2007; 
Morgan, Farkas, & Wu, 2009). Thus, it is clear that the acquisition of early mathematics 
knowledge by all children must be a major educational priority. 

To address this educational priority, our research group has developed an early 
mathematics intervention, Pre-K Mathematics (Klein & Starkey, 2004). With collaborators, we 
have rigorously evaluated it over multiple studies and have found it to be highly effective at 
improving mathematical knowledge in economically disadvantaged pre -kindergarten children 
relative to a control group (Klein, et al., 2008; also see What Works Clearinghouse website, 
http://ies.ed.gov/ncee/wwc). However, testing the scale-up of an early mathematics intervention 
already shown to have beneficial effects shifts the focus from the effectiveness of the curriculum 
per se to the effectiveness of the proposed process by which public preschool and Head Start 
programs can be assisted to implement the curriculum well enough to attain its benefits. The 
level of scale used in this project - school district preschool programs and grantee-operated Head 
Start programs - is the customary level at which preschool curricular adoptions are made. 

Our scale-up project was comprised of two studies. First, a Main Study was conducted to 
evaluate the effectiveness of Pre-K Mathematics when implemented at scale. Second, a 
Sustainability Study was conducted to determine whether programs could sustain an effective 
implementation of Pre-K Mathematics beyond the initial year in which intensive professional 
development was provided. In this presentation, we will review the principal child outcomes and 
mediation model from the Main Study, and then focus on the Sustainability Study and a 
comparison between the findings of the two studies. Finally, we will discuss the policy 
implications of these findings for establishing partnerships with preschool programs in order to 
help them implement and sustain effective early mathematics curricula. 


SREE Fall 2012 Conference Abstract Template 


1 



Main Study 

Purpose / Objective / Research Question / Focus of Study: 

The Main Study of the Pre-K Mathematics Scale-Up Project was a randomized controlled field 
trial that had two principal objectives: (1) to determine whether and how the Pre-K Mathematics 
intervention continues to be effective when implemented at a customary level of scale and at a 
distance from the developer, and (2) to document the process of implementing the math 
intervention in varied contexts in order to understand under what conditions the intervention is 
effective. 

Setting: 

The Main Study was conducted in varied contexts, which included Head Start and state-funded 
public preschool programs in California, Kentucky, and Indiana. Teachers from 94 classrooms 
at 63 school sites (Head Start Centers or public preschools) were included in the experiment. 

Population / Participants / Subjects: 

The sample for the experimental field trial included 744 children. The California sample was 
comprised of ethnically diverse, low-income urban children. The Kentucky/Indiana sample was 
comprised of predominantly Caucasian, low-income rural children. Attrition during the pre-k 
year was 10.1%. 

Intervention / Program / Practice: Pre-K Mathematics (Klein & Starkey, 2004), the curricular 
intervention used in this project, included classroom, home, and professional development 
components. The classroom component provided conceptually broad support for the 
development of children’s informal mathematical knowledge. It consisted of a set of 27 small- 
group math activities with concrete manipulatives, math software, and a math learning center in 
the classroom. The mathematical content of the small-group activities was based on based on 
developmental research about the nature and extent of early mathematical knowledge. Teachers 
typically conducted small-group math activities twice a week with groups of 4 children for 20 
minutes, plus a review day for absent or struggling children. The home component of the 
intervention provided parents with activities to support their children’s mathematical 
development and complemented the math support children were receiving at preschool. A 
Spanish version of the home math activities was used with Spanish-speaking families. Finally, 
the professional development component utilized a trainer-of-trainers model to implement the 
math curriculum on the scale of a Head Start program or school district preschool program. 
Program trainers or coaches attended a Facilitators Institute to learn Pre-K Mathematics as well 
as to learn how to conduct fidelity visits and support teachers implementing the math curriculum 
in their classrooms. The program trainers, in turn, helped train the teachers and monitored their 
fidelity of implementation of the math curriculum in their classrooms. 

Research Design: 

The basic research design was a cluster randomization in which 94 classrooms at 63 sites were 
randomly assigned to the intervention and control conditions in accordance with the prior power 
analysis. If there were more classrooms or sites than needed in any grouping, the appropriate 
numbers were selected randomly. 

Data Collection and Analysis: 

A set of instruments was used to assess children’s early mathematical knowledge. The Child 
Math Assessment (CMA) was used to provide a measure of children’s infonnal mathematical 
knowledge across a broad range of skills and concepts. The TEMA-3 was used along with the 
CMA as a standardized measure of children’s developing mathematical knowledge. Classroom 
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observation (EMCO) and curriculum fidelity instruments were also administered to collect data 
on potential mediators and moderators of intervention effects. 


Findings / Results: 

The fidelity of teachers’ implementation of Pre-K Mathematics small-group activities was 
measured 6 times across the school year. Observed fidelity was high both in California (.92) and 
in Kentucky/Indiana (.90). This indicates that, in general, teachers implemented small-group 
math activities as intended by the developers. Children’s mathematical knowledge was assessed 
at pretest and posttest of the pre-k year using the CMA and TEMA-3. Nested analyses found that 
significantly greater gains in mathematical knowledge were experienced by treatment children, 
relative to control children, on both the CMA (ES=.83) and the TEMA-3 (ES=.45). Moreover, 
the intervention effect was consistent across states and program types. There was no significant 
effect of state or program type, and no significant interaction of time with state or program type. 
This indicates that the Pre-K Mathematics intervention was robust across the varied preschool 
contexts included in this study. 

Classroom observations of teachers’ math practices were conducted in both intervention and 
control classrooms. Analyses revealed that treatment teachers, relative to control teachers, 
learned to use a set of best math practices - small-group activities that were mathematically 
focused and scaffolded by teachers. Consistent with the proposed mediation model, it was found 
that this set of best math practices indirectly mediated the intervention by engaging children’s 
developing mathematical cognition. 

Sustainability Study 

Purpose / Objective / Research Question / Focus of Study: 

The Sustainability Study was conducted in the same intervention classrooms with the same 
teachers during the year following the Main Study. The principal objective was to determine 
whether teachers were able to sustain implementation of the math intervention in an effective 
manner beyond the initial year in which they received professional development. 

Setting: 

This study was conducted in the same Head Start and state-funded preschool programs in 
California and Kentucky/Indiana as in the Main Study. 

Population / Participants / Subjects: 

The sample included 326 children in 39 treatment classrooms. The California sample was 
comprised of low-income, ethnically diverse, urban children. The Kentucky/Indiana sample was 
comprised of low-income, predominantly Caucasian, rural children. Attrition during the pre-k 
year was 10%. 

Intervention / Program / Practice: 

Intervention teachers from the Main Study continued implementing the classroom and home 
components of Pre-K Mathematics with pre-k children during the Sustainability year. Fidelity 
was similar in the first and second years of implementation, .91 and .93, respectively. Teachers 
also provided children with similar amounts of the intended dosage of classroom activities in the 
first and second years of implementation, 85% and 88%, respectively, and children mastered 
similar percentages of small group activities, 71% and 65%, respectively. Child absences were 
the most common reason for missing doses. 

Research Design: 

The randomization for the Main Study assigned 48 classrooms and their teachers to the 
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intervention condition. Of these, 39 teachers continued teaching in their programs at the pre-k 
level during the subsequent year in which we conducted the Sustainability Study. The remaining 
9 teachers could not be included, because they were no longer teaching pre-k children in the 
participating programs. 

Data Collection and Analysis: 

The same set of instruments was used in the Sustainability Study in order to determine the extent 
to which the implementation and positive child outcomes could be sustained by programs. 

Findings / Results: 

Child math outcomes. 

Children’s mathematical knowledge in pre-kindergarten was assessed at pretest and posttest using 
the CMA and TEMA-3. A 3-level ANOVA of CMA scores, with children nested within 
classrooms within sites, revealed no significant difference in pretest scores between conditions. A 
significant Condition X Time interaction, F(l,527)=68.82,/?<.0001 (ES=.70), indicated greater 
change in intervention children’s scores across time. A parallel set of analyses of children’s 
TEMA-3 scores at pretest and posttest revealed a significant Condition X Time interaction 
favoring treatment children, F=(l, 527) = 34.57, p<.0001 (ES=.45). 

Comparison of child math outcomes between teachers’ first and second year of 
implementation. The next set of analyses compared gains in children’s math knowledge 
between treatment teachers’ first year of implementation (Main Study) and their second year of 
implementation (Sustainability Study). There were no significant differences between 
implementation years in children’s CMA scores or TEMA scores. 

Kindergarten follow up: TEMA. Longitudinal follow-up analyses examined whether 
the positive effects of the intervention on children’s math knowledge persisted into kindergarten. 
TEMA-3 raw scores at pre-k pretest (time 1), pre-k posttest (time 2), and middle of grade K 
(time 3) were analyzed using a repeated measures mixed model. Treatment children, relative to 
control children, demonstrated significantly greater mathematical knowledge both at time 2, F(\, 

1398) = 20.37,/? < .0001, and time 3, F(l,1398) = 38.44,/? < .0001. 

Kindergarten follow up: Teacher report cards. Kindergarten teachers, who were blind 
to children’s assigned condition during the pre-k intervention year, rated children's mathematical 
ability on a Unifonn Report Card. These data provide further evidence that the effects of the 
intervention persisted into kindergarten. Teachers reported that mathematical knowledge was 
more extensive in treatment children than in control children, F(l,547)=2 1.245,/? < .0001. 

Teachers’ mathematics practices. 

In the first year of implementation, it was found that treatment teachers, relative to control 
teachers, spent more time on math instruction overall and more time engaged in instruction that utilized 
a set of best math practices, which had been emphasized in the Pre-K Mathematics training provided. 
This set of best practices included greater use of small-group math activities, focal (mathematically 
explicit) activities, and scaffolding. In the second year of implementation, treatment teachers, relative to 
control teachers, continued to spend more time engaged in math instruction compared to control 
teachers, F(l,77) = 19.394,/? < .0001. Treatment teachers also continued to use the set of best math 
practices. Specifically, they devoted more instructional time on small-group math activities, F(l,77) = 
74.176,/? < .0001, focal math activities, F(l,77) = 24.305,/? < .0001, and provided children with 
scaffolding for math learning more often, F(\,77) = 26.040,/? < .0001 compared to control teachers. As 
in the first year of implementation, there were no significant differences by condition in the amount of 
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instructional time devoted to whole-group math activities, embedded math activities, or math activities 
in which children were not provided with scaffolding. 


Conclusions: 

Public Preschool Programs Can Improve Children’s Math Outcomes by Implementing an 
Effective Math Curriculum Coupled with High Quality Mathematics Practices 

A policy implication that can be drawn from these findings is that programs need both an 
effective math curriculum and teachers who utilize a set of best mathematics practices when 
implementing the curriculum. It is this combination that helps public preschool programs address 
the SES-related math gap. Furthennore, significant improvement in children’s math outcomes 
can be obtained in the initial year of implementation as evidenced by the Main Study results. 
Public Preschool Programs Can Sustain High Quality Instructional Practices 

In the sustainability year, treatment teachers were able to continue to implement Pre-K 
Mathematics with a high degree of fidelity and to deliver a good dosage of small-group math 
activities in their classrooms. With support from program trainers, teachers sustained their use of 
a set of best mathematics practices that had been emphasized in their professional development 
training, and continued to monitor children’s math learning across the pre-kindergarten year. 
Public Preschool Programs Can Sustain an Effective Implementation with Support from 
Program Trainers 

The principal hypothesis - that the math intervention would lead to increased 
mathematical knowledge in participating children by the end of the pre-kindergarten year - was 
clearly supported. Furthennore, treatment effects, as measured by a standardized assessment and 
by children’s kindergarten teachers, persisted into the kindergarten year. Thus, programs were 
able to sustain an effective implementation when implementing at a customary, program-wide 
level of scale. 
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